System and method for a tracked procedure

ABSTRACT

Disclosed is a navigation system. The navigation system may be used to at least assist in a procedure. The system may assist in delineating objects and/or determining physical boundaries of image elements. The system may assist in planning and/or a workflow of the procedure.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application includes subject matter related to U.S. Pat. App. Ser.No. 15/965,320. The entire disclosure(s) of (each of) the aboveapplication(s) is (are) incorporated herein by reference.

FIELD

The subject disclosure relates generally to a system and method fordetermining a position, including location and orientation, of a memberin space relative to a subject and identifying features in an image andworkflow efficiency.

BACKGROUND

This section provides background information related to the presentdisclosure which is not necessarily prior art.

In a navigation system for various procedures, such as surgicalprocedures, assembling procedures, and the like, an instrument or objectmay be tracked. The instrument may be tracked by one or more trackingsystems of various operation modes, such as by measuring an effect of amagnetic field on a sensor coil and/or determining a location withoptical sensors. The sensor coil may include a conductive material thatis placed within a magnetic field where a current is induced in thesensor coil. The measured induced current may be used to identify ordetermine a position of the instrument or object.

The electro-magnetic field may be generated with a plurality of coils,such as three orthogonally placed coils. Various transmitter or fieldgeneration systems include the AxiEM™ electro-magnetic navigation systemsold by Medtronic Navigation, Inc., having a place of business inLouisville, Colo. The AxiEM™ electro-magnetic navigation system mayinclude a plurality of coils that are used to generate anelectro-magnetic field that is sensed by a tracking device, which may bethe sensor coil, to allow a navigation system, such as a StealthStation®surgical navigation system, to be used to track and/or illustrate atracked position of an instrument.

The tracking system may also, or alternatively, include an opticaltracking system. Optical tracking systems include those such as theStealthStation® S7® tracking system. The optical tracking systemincludes a set of cameras with a field of vision to triangulate aposition of the instrument.

SUMMARY

This section provides a general summary of the disclosure, and is not acomprehensive disclosure of its full scope or all of its features.

A system for performing a procedure is disclosed. The procedure may alsobe performed on a living subject such as an animal, human, or otherselected patient. The procedure may include any appropriate type ofprocedure, such as one being performed on an inanimate object (e.g. anenclosed structure, airframe, chassis, etc.). Nevertheless, theprocedure may be performed using a navigation system where a trackingsystem is able to track a selected one or more items.

A navigation system may be used to navigate an instrument relative to asubject for performing a procedure. In various embodiments, theprocedure may include a procedure on a spine such as a spinal fusionwhere two or more vertebrae are connected together with a selectedimplant system or assembly. The implant system may include more than onecomponent that is interconnected at a selected time. Positioning of aportion of the implant system, such as a screw, may be performedrelative to a boney structure including a vertebrae. The screw may bepositioned into the vertebrae along a selected trajectory and to aselected depth along the trajectory into the vertebrae. In addition tothe above example, other appropriate procedures may also be performedrelative to and/or on the spine or other appropriate locations.

At a selected time, such as for performing a procedure and/or planning aprocedure, image data may be acquired of the subject. Image data may beused to generate an image that is displayed on the display device. Theimage data may include any appropriate image data such as computedtomography image data, magnetic resonance image data, X-ray cone beamimage data (such as with a x-ray cone beam imager). Further, the imagermay be any appropriate imager such as the O-Arm® imaging system, asdiscussed further herein. A selected set of instructions, such as acomputer vision algorithm, may be used to identify portions within theimage data, such as individual vertebrae. The instructions may include amachine learning technique or process, such as a neural network system,that is programed to determine the boundaries of the vertebrae. Theimage data may be analyzed substantially or entirely automaticallywithin the neural network to determine the boundaries of the vertebrae.

A selected workflow may be used to efficiently and effectively perform aprocedure. The workflow may include analysis or reference to the imagedata to determine and/or segment selected portions or features in theimage, such as segmenting specific vertebrae. The workflow may be usedto operate the navigation system in an automatic manner to provideinformation to a user, such as a clinician or a surgeon, during theperformance of the procedure. The image data, having identifiedboundaries of selected features (e.g. vertebra or vertebra portions),may assist or allow the system in automatically identifying a trajectoryfor performing a procedure, a specific implant for positioning relativeto specific vertebrate, and other portions of the procedure.Accordingly, a workflow may be automated or have selected userinteraction, such as reduced or faster, in performing selected and/orstandard portions of a selected procedure.

Further areas of applicability will become apparent from the descriptionprovided herein. The description and specific examples in this summaryare intended for purposes of illustration only and are not intended tolimit the scope of the present disclosure.

DRAWINGS

The drawings described herein are for illustrative purposes only ofselected embodiments and not all possible implementations, and are notintended to limit the scope of the present disclosure.

FIG. 1 is an environmental view of a navigation system;

FIG. 2 is a schematic flowchart of a segmentation process;

FIG. 3 is a schematic view of the convolution operator

FIG. 4 is a schematic view of a CNN, according to various embodiments;

FIG. 5 is a flowchart for training a CNN, according to variousembodiments;

FIG. 6 is a flowchart for a segmentation of an image with a previouslytrained CNN, according to various embodiments;

FIG. 7A and FIG. 7B is a flowchart for a workflow and operation of asurgical navigation system, according to various embodiments;

FIG. 8 is an environmental and display view of tracking and displayingan instrument projection;

FIG. 9A is a display view of a proposed or planned procedure with aninstrument projection;

FIG. 9B is a display view of a partially complete planned procedure withthe instrument projection;

FIG. 9C is a display view of a complete planned procedure and a reverseprojection; and

FIG. 10 is a display view of a reverse projection as a plan and atracked view of an implant.

Corresponding reference numerals indicate corresponding parts throughoutthe several views of the drawings.

DETAILED DESCRIPTION

Example embodiments will now be described more fully with reference tothe accompanying drawings.

With initial reference to FIG. 1, a navigation system 10 is illustrated.The navigation system 10 may be used for various purposes or proceduresby one or more users, such as a user 12. The navigation system 10 may beused to determine or track a position of an instrument 16 in a volume.The position may include both a three dimensional X,Y,Z location andorientation. Orientation may include one or more degree of freedom, suchas three degrees of freedom. It is understood, however, that anyappropriate degree of freedom position information, such as less thansix-degree of freedom position information, may be determined and/orpresented to the user 12.

Tracking the position of the instrument 16 may assist the user 12 indetermining a position of the instrument 16, even if the instrument 16is not directly viewable by the user 12. Various procedures may blockthe view of the user 12, such as performing a repair or assembling aninanimate system, such as a robotic system, assembling portions of anairframe or an automobile, or the like. Various other procedures mayinclude a surgical procedure, such as performing a spinal procedure,neurological procedure, positioning a deep brain simulation probe, orother surgical procedures on a living subject. In various embodiments,for example, the living subject may be a human subject 20 and theprocedure may be performed on the human subject 20. It is understood,however, that the instrument 16 may be tracked and/or navigated relativeto any subject for any appropriate procedure. Tracking or navigating aninstrument for a procedure, such as a surgical procedure, on a human orliving subject is merely exemplary.

Nevertheless, in various embodiments, the surgical navigation system 10,as discussed further herein, may incorporate various portions orsystems, such as those disclosed in U.S. Pat. Nos. RE44,305; 7,697,972;8,644,907; and 8,842,893; and U.S. Pat. App. Pub. No. 2004/0199072, allincorporated herein by reference. Various components that may be usedwith or as a component of the surgical navigation system 10 may includean imaging system 24 that is operable to image the subject 20, such asan O-Arm® imaging system, magnetic resonance imaging (MRI) system,computed tomography system, etc. A subject support 26 may be used tosupport or hold the subject 20 during imaging and/or during a procedure.The same or different supports may be used for different portions of aprocedure.

In various embodiments, the imaging system 24 may include a source 24 s.The source may emit and/or generate X-rays. The X-rays may form a cone24 c, such as in a cone beam, that impinge on the subject 20. Some ofthe X-rays pass though and some are attenuated by the subject 20. Theimaging system 24 may further include a detector 24 d to detect theX-rays that are not completely attenuated, or blocked, by the subject20. Thus, the image data may include X-ray image data. Further, theimage data may be two-dimensional (2D) image data.

Image data may be acquired, such as with one or more of the imagingsystems discussed above, during a surgical procedure or acquired priorto a surgical procedure for displaying an image 30 on a display device32. In various embodiments, the acquired image data may also be used toform or reconstruct selected types of image data, such asthree-dimensional volumes, even if the image data is 2D image data. Theinstrument 16 may be tracked in a trackable volume or a navigationalvolume by one or more tracking systems. Tracking systems may include oneor more tracking systems that operate in an identical manner or moreand/or different manner or mode. For example, the tracking system mayinclude an electro-magnetic (EM) localizer 40, as illustrated in FIG. 1.In various embodiments, it is understood by one skilled in the art, thatother appropriate tracking systems may be used including optical, radar,ultrasonic, etc. The discussion herein of the EM localizer 40 andtracking system is merely exemplary of tracking systems operable withthe navigation system 10. The position of the instrument 16 may betracked in the tracking volume relative to the subject 20 and thenillustrated as a graphical representation, also referred to as an icon,16 i with the display device 32. In various embodiments, the icon 16 imay be superimposed on the image 30 and/or adjacent to the image 30. Asdiscussed herein, the navigation system 10 may incorporate the displaydevice 30 and operate to render the image 30 from selected image data,display the image 30, determine the position of the instrument 16,determine the position of the icon 16 i, etc.

With reference to FIG. 1, the EM localizer 40 is operable to generateelectro-magnetic fields with a transmitting coil array (TCA) 42 which isincorporated into the localizer 40. The TCA 42 may include one or morecoil groupings or arrays. In various embodiments, more than one group isincluded and each of the groupings may include three coils, alsoreferred to as trios or triplets. The coils may be powered to generateor form an electro-magnetic field by driving current through the coilsof the coil groupings. As the current is driven through the coils, theelectro-magnetic fields generated will extend away from the coils 42 andform a navigation domain or volume 50, such as encompassing all or aportion of a head 20 h, spinal vertebrae 20 v, or other appropriateportion. The coils may be powered through a TCA controller and/or powersupply 52. It is understood, however, that more than one of the EMlocalizers 40 may be provided and each may be placed at different andselected locations.

The navigation domain or volume 50 generally defines a navigation spaceor patient space. As is generally understood in the art, the instrument16, such as a drill, lead, etc., may be tracked in the navigation spacethat is defined by a navigation domain relative to a patient or subject20 with an instrument tracking device 56. For example, the instrument 16may be freely moveable, such as by the user 12, relative to a dynamicreference frame (DRF) or patient reference frame tracker 60 that isfixed relative to the subject 20. Both the tracking devices 56, 60 mayinclude tracking portions that are tracking with appropriate trackingsystems, such as sensing coils (e.g. conductive material formed orplaced in a coil) that senses and are used to measure a magnetic fieldstrength, optical reflectors, ultrasonic emitters, etc. Due to thetracking device 56 connected or associated with the instrument 16,relative to the DRF 60, the navigation system 10 may be used todetermine the position of the instrument 16 relative to the DRF 60.

The navigation volume or patient space may be registered to an imagespace defined by the image 30 of the subject 20 and the icon 16 irepresenting the instrument 16 may be illustrated at a navigated (e.g.determined) and tracked position with the display device 32, such assuperimposed on the image 30. Registration of the patient space to theimage space and determining a position of a tracking device, such aswith the tracking device 56, relative to a DRF, such as the DRF 60, maybe performed as generally known in the art, including as disclosed inU.S. Pat. Nos. RE44,305; 7,697,972; 8,644,907; and 8,842,893; and U.S.Pat. App. Pub. No. 2004/0199072, all incorporated herein by reference.

The navigation system 10 may further include a navigation processorsystem 66. The navigation processor system 66 may include the displaydevice 32, the TCA 40, the TCA controller 52, and other portions and/orconnections thereto. For example, a wire connection may be providedbetween the TCA controller 52 and a navigation processing unit 70.Further, the navigation processor system 66 may have one or more usercontrol inputs, such as a keyboard 72, and/or have additional inputssuch as from communication with one or more memory systems 74, eitherintegrated or via a communication system. The navigation processorsystem 66 may, according to various embodiments include those disclosedin U.S. Pat. Nos. RE44,305; 7,697,972; 8,644,907; and 8,842,893; andU.S. Pat. App. Pub. No. 2004/0199072, all incorporated herein byreference, or may also include the commercially availableStealthStation® or Fusion™ surgical navigation systems sold by MedtronicNavigation, Inc. having a place of business in Louisville, Colo.

Tracking information, including information regarding the magneticfields sensed with the tracking devices 56, 60, may be delivered via acommunication system, such as the TCA controller, which also may be atracking device controller 52, to the navigation processor system 66including the navigation processor 70. Thus, the tracked position of theinstrument 16 may be illustrated as the icon 16 i relative to the image30. Various other memory and processing systems may also be providedwith and/or in communication with the processor system 66, including thememory system 72 that is in communication with the navigation processor70 and/or an imaging processing unit 76.

The image processing unit 76 may be incorporated into the imaging system24, such as the O-Arm® imaging system, as discussed above. The imagingsystem 24 may, therefore, include various portions such as a source anda x-ray detector that are moveable within a gantry 78. The imagingsystem 24 may also be tracked with a tracking device 80. It isunderstood, however, that the imaging system 24 need not be presentwhile tracking the tracking devices, including the instrument trackingdevice 56. Also, the imaging system 24 may be any appropriate imagingsystem including a MRI, CT, etc.

In various embodiments, the tracking system may include an opticallocalizer 82. The optical localizer 82 may include one or more camerasthat view or have a field of view that defines or encompasses thenavigation volume 50. The optical localizer 82 may receive light (e.g.infrared or ultraviolet) input to determine a position or track thetracking device, such as the instrument tracking device 56. It isunderstood that the optical localizer 82 may be used in conjunction withand/or alternatively to the EM localizer 40 for tracking the instrument16.

Information from all of the tracking devices may be communicated to thenavigation processor 70 for determining a position of the trackedportions relative to each other and/or for localizing the instrument 16relative to the image 30. The imaging system 24 may be used to acquireimage data to generate or produce the image 30 of the subject 20. It isunderstood, however, that other appropriate imaging systems may also beused. The TCA controller 52 may be used to operate and power the EMlocalizer 40, as discussed above.

The image 30 that is displayed with the display device 32 may be basedupon image data that is acquired of the subject 20 in various manners.For example, the imaging system 24 may be used to acquire image datathat is used to generate the image 30. It is understood, however, thatother appropriate imaging systems may be used to generate the image 30using image data acquired with the selected imaging system. Imagingsystems may include magnetic resonance imagers, computed tomographyimagers, and other appropriate imaging systems. Further the image dataacquired may be two dimensional or three dimensional data and may have atime varying component, such as imaging the patient during a heartrhythm and/or breathing cycle.

In various embodiments, the image data is a 2D image data that isgenerated with a cone beam. The cone beam that is used to generate the2D image data may be part of an imaging system, such as the O-Arm®imaging system. The 2D image data may then be used to reconstruct a 3Dimage or model of the imaged subject, such as the patient 20. Thereconstructed 3D image and/or an image based on the 2D image data may bedisplayed. Thus, it is understood by one skilled in the art that theimage 30 may be generated using the selected image data.

Further, the icon 16 i, determined as a tracked position of theinstrument 16, may be displayed on the display device 32 relative to theimage 30. In addition, the image 30 may be segmented, for variouspurposes, including those discussed further herein. Segmentation of theimage 30 may be used determine and/or delineate objects or portions inthe image. The delineation may include or be made as a mask that isrepresented on a display. The representation may be shown on the displaysuch as with a graphical overlay of a mask, which may also be referredto as an icon. The icon may the segmented mask and may not be simplifiedin any manner. In various embodiments, the delineation may be used toidentify boundaries of various portions within the image 30, such asboundaries of one or more structures of the patient that is imaged, suchas the vertebrae 20 v. Accordingly, the image 30 may include an image ofone or more of the vertebrae 20 v, such as a first vertebrae 20 vi and asecond vertebrae 20 vii. As discussed further herein, the vertebrae,such as the first and second vertebrae 20 vi, 20 vii may be delineatedin the image which may include and/or assist in determining boundariesin images, such as 3D and 2D images. In various embodiments, thedelineation may be represented such as with an icon 20 vi′ or a secondicon 20 vii′. The boundaries 20 vi′, 20 vii′ may be determined in anappropriate manner and for various purposes, as also discussed furtherherein. Further, the icon may be used to represent, for display, aselected item, as discussed herein, including the delineation of theobject, boundary, etc.

According to various embodiments, the image 30 may be segmented in asubstantially automatic manner. In various embodiments, the automaticsegmentation may be incorporated into a neural network, such as aconvolutional neural network (CNN). The CNN may be taught or learn todetermine, such as with a probability or prediction, various features,according to various embodiments. Various features may include objects(e.g. vertebra) or parts or portions of objects (e.g. pedicle), andsegmentations or boundaries of these objects or portions. The selectedsegmentations may include identifying a segmentation of selectedvertebrae, such as the first vertebrae 20 vi and the second vertebrae 20vii. The selected segmentation may be displayed with a selectedgraphical representation such as a segmentation icon or representation20 vi′ and 20 vii′ for display on the display device 32.

The icons are displayed alone on the display 32 and/or superimposed onthe image 30 for viewing by a selected user, such as the user 12 whichmay be a surgeon or other appropriate clinician. Moreover, onceidentified, the boundaries or other appropriate portion, whetherdisplayed as icons or not, may be used for various purposes. Theboundaries may identify a physical dimension of the vertebrae, positionsof the vertebrae in space (i.e. due to registration of the image 30 tothe subject 20 as discussed above), possible identified trajectories(e.g. for implantation placement), or the like. Therefore, the image 30may be used in planning and/or performing a procedure whether the icons20 vi′, 20 vii′ are displayed or the geometry of the boundaries is onlydetermined and not displayed as an icon.

Turning reference to FIG. 2, a process or method for identifying aportion of an image, also referred to as segmenting an image, isillustrated in the flowchart 100. The flowchart 100, is a generalflowchart and a more specific process, such as a CNN, will be discussedin further detail herein. Generally, however, the segmentation processbegins with an input of image data. The image data may include anyappropriate image data such as computed tomography image data, magneticresonance image data, X-ray cone beam image data. Further, the imagermay be any appropriate imager such as the O-Arm® imaging system, asdiscussed herein. The O-Arm® imaging system may be configured to acquireimage data for a 360 degrees around a subject and include 2D image dataand/or a 3D reconstruction based on the 2D image data. Further, theO-Arm® imaging system may generate images with a x-ray cone beam.

The image data may include 2D image data or a 3D model reconstructedfrom the 2D image data in block 104. The 2D image data or thereconstructed 3D image data may be from an imaging system such as theimaging system 24. The imaging system 24, as discussed above, mayinclude the O-Arm® imaging system. The imaging system 24 may generate aplurality of two dimensional image data that may be used to reconstructa three dimensional model of the subject 20 including one or more of thevertebrae 20 v. The input image data may also be acquired at anyappropriate time such as during a diagnostic or planning phase ratherthan in an operating theatre, as specifically illustrated in FIG. 1.Nevertheless, the image data may be acquired of the subject 20 with theimaging system 24 and may be input or accessed in block 104.

The image data acquired with the imaging system 24 may be of a selectedimage quality that may be difficult to identify various boundaries ofimage portions, such as the vertebrae 20 v. Nevertheless, as discussedfurther herein, a neural network may be used to automatically identifythe boundaries of the imaged portions to segment the image data.

The image data from block 104 may be processed with a selected system,such as a neural network or an artificial neural network, in block 106.The artificial neural network (ANN) may be a selected appropriate typeof artificial neural network such as a convolutional neural network(CNN). The CNN may be taught or learn to analyze the input image datafrom block 104 to segment selected portions of the image data. Forexample, as discussed above, the CNN in block 106 may be used toidentify boundaries of vertebral bodies in the image data from block104. As discussed above the boundaries of the vertebral bodies may bedisplayed on the display device 32 either alone and/or in combinationwith the image 30. Accordingly, output segmented image data or outputsegmented data may be made in block 110. The outputted segmented datamay be stored in a selected memory system, such as the navigation memory74 or a segmented image memory 112 (See FIG. 1). The output segmentedimage data may segment selected portions, such as the vertebrae 20 v asdiscussed above, for various purposes.

Accordingly, the flowchart 100 can start in block 102 and then access orinput image data in block 104 to output segmented image data (and/orsegmented masks) in block 110 and display or store the segmented imagedata in block 114. The process may then end in block 118 and/or allowfor further processing or workflow, as discussed further herein. It isunderstood that the selected portions of the flowchart or process 100,however, may include a plurality of additional steps in addition tothose discussed above. For example, the CNN may be developed and thentaught to allow for an efficient and/or fast segmentation of a selectedportion of the image data that is accessed or inputted from block 104.The segmentation may be a specific, such as identifying the vertebrae,or general such as identifying selected boundaries or changing contrastin the image data.

Turning reference to FIG. 3, the CNN used in block 106 may be developedand taught, as briefly discussed above, and discussed in further detailherein. The CNN is based upon generally known convolutional neuralnetwork techniques such as that disclosed in Ozgon Qigek, AhmedAbdulkadir, Soeren S. Lienkamp, Thomas Brox, Olaf Ronneberger, “3DU-Net: Learning Dense Volumetric Segmentation from Sparse Annotation”,International Conference on Medical Image Computing andComputer-Assisted Intervention, Springer, Cham, pp. 424-432 (2016)(https://arxiv.org/pdf/1606.06650.pdf (2016)), incorporated herein byreference. The CNN may be developed to efficiently identify selectedportions of images by analyzing the image data and causing an excitationof an artificial neuron to make a judgment or calculation of a selectedlearned portion in a new input. According, the input image data fromblock 104 may be analyzed or processed with the CNN in block 106 basedupon a teaching of the CNN, as discussed further herein.

With continued reference to FIG. 3, the CNN generally allows for theteaching of the CNN to identify image features in image data, such asthe accessed image data in block 104. For example, a kernel K or filterof a selected dimension that may be previously defined and/or learned bythe CNN. The kernel or filter K may then be applied to the image I in astepwise manner, such as moving the filter one pixel or one voxel at atime. The filter K is a single component in the CNN which may consistsof hundreds or thousands of interconnected filters arranged in layers.As discussed herein, the first layer filters operate on the image I,filters on the next layers operate on the output of the previous layers.That is, as illustrated in FIG. 3, the kernel K of a selected size maybe moved stepwise a selected dimension, such as one pixel or voxel,throughout the entire image. The filter K may be learned and used toidentify a selected portion or an “interesting” portion of the image. Asillustrated in FIG. 3, the filter or kernel (K) is applied to an image(for example a two dimensional image). A product, or summation ofproducts (such as a dot product of the kernel K and portion of the imageI) may then be saved or stored for a further layer as a convolutionalproduct matrix I*K. The product matrix, will have a dimensions less thanthe input given the size of the filter and the stride selected. Thesummation in a two dimensional manner is illustrated or defined byequation one (Eq.1):

$\begin{matrix}{\left( {I*K} \right)_{xy} = {\sum\limits_{i = 1}^{h}{\sum\limits_{j = 1}^{w}{K_{ij} \cdot I_{{x + i - 1},{y + j - 1}}}}}} & {{Eq}.\mspace{14mu} 1}\end{matrix}$Eq. 1 includes the image data I (i.e. includes pixels or voxels) in aselected array. The K represents the kernel or filter where the filterhas a height and width (i.e. in a two dimensional kernel relating to twodimensional image data) and the output I*K of convolutional matrix isthe summation dot product of the input image I and the kernel Kaccording to Eq. 1.

The convolution, as discussed further herein, includes moving the kernelover the input image to generate an activation map, of which the I*K isa part. As illustrated in FIGS. 3 and 4, an activation map layer isformed by the application of each filter to the image. A plurality ofthe filters, which may be learned by the CNN, may then be stacked toproduce the output volume (e.g. for three dimensional image data) thatgenerates a segmentation of the input image data. Thus, the CNN mayoutput a three-dimensional (3D) output image or model that includes oris a segmentation of the image data.

Various features may be incorporated into the CNN, including those asknown in the art and/or additional features to assist in determining andcreating an efficient segmentation of a selected image. For example anamount of connectivity may include a local connectivity that isequivalent to the selected filter or kernel size. It is understood thatkernel size may be selected based upon a resolution of an input image, aprocessing speed selected, or the like. In various embodiments thekernel size may be selected to be about 3×3×3 in size, such as pixeldimensions. It is understood, however, that different kernel sizes maybe selected.

A size of an output may also be dependent upon various parameters thatare selected to choose or select an output volume. For example, variousparameters may include depth, stride, and a zero-padding or anyappropriate padding. Depth includes a number of distinct differentkernels that are convolved in a layer. For example, as illustrated inFIG. 3 the kernel includes a selected array of convolution operations(i.e. an array of ones and zeros). It is understood that a plurality ofdifferent filters may be applied to each layer in a CNN where the kernelincludes different operational arrays. A stride refers to the number ofelements (e.g. pixels, voxels, or other selected image element) thatdetermine the amount of movement of the filter within the input volumeper step. Padding may further be added to an input image in anyparticular layer to account for the decrease in an output image side dueto the striding of a kernel within an image. As illustrated in FIG. 3,moving the kernel with a stride equal to one pixel will decrease theoutput matrix by two pixel dimension (e.g. an input of 7×7 having akernel 3×3 with a stride of 1 will output a matrix of 5×5). Padding theinput to include zero image data or pixels can maintain the output sizeto be equal to the input size of the volume or image.

In the CNN, in addition to the convolution including the size of thefilter and features of the filter, as discussed above, additionaloperations may also occur. For example, in the CNN a pooling layer maybe added to down sample the output. For example a pooling, such as amax-pooling, operation may attempt to reduce the number of parametersand reduce or control over fitting. The max pooling may identify orselect only the maximum volume (e.g. a maximum pixel or voxel value)within a filter size for an output. For example, a max pooling filtermay include a 2×2 filter that is applied in a stride of two along aselected dimension, such as two dimensional image for a two dimensionalimage. The max pooling will take only the maximum valued pixel from thefilter area to the output.

Additional operations may also include batch normalization such as thatdescribed in Sergey Ioffe, Christian Szegedy, “Batch Normalization:Accelerating Deep Network Training by Reducing Internal CovariateShift”, ICML, 2015 (http://arxiv.orq/abs/1502.03167 (2015)),incorporated herein by reference. The batch normalization may be appliedat selected points or layers in the CNN, such as an initial layer orafter each convolutional layer. The batch normalization may cause orwill cause an activation throughout the CNN to achieve a selecteddistribution, such as a unit Gaussian distribution, at a selected pointin the training, such as a beginning point in the training. Batchnormalization allows increased depth of the network, acceleratedtraining, and robustness to initialization.

An output from a neuron in a layer may include calculating a weightedsum and adding a bias to an activation function is then applied to theinput and bias to produce an output. The weights from a neuron or layeroutput may be further analyzed or incorporated into the CNN. The outputfrom neurons, to achieve the selected output, may have a weighted lossfunction to assist in activating or reducing an influence of a selectedactivation function of a neuron. A weighted loss function givesdifferent importance to different labels based on certain properties,e.g. boundary of an object vs non-boundary

An output from a neuron includes the activation function that is arectified linear unit function except for an output layer function thatis a soft max activation function. The activation function may include afunction such as softmax function, or normalized exponential function asis generally known in the art. A rectified linear units function is afunction generally defined as f(x)=max(0,x). Accordingly, the rectifiedlinear units function can provide an activation of a neuron in the CNNwhen the rectified linear unit activation function is greater than aselected threshold. In the rectified function only a positive componentof the function is compared to the threshold. In an output layer, themask or delineation of a selected portion of the input image identifiedas a selected object (also referred to as a label, e.g. a vertebra orportion thereof) is determined if the output probability map is above aselected threshold probability, such as 35%. In various embodiments, themask or delineation of the portion of the input image identified as aselected object is the label with the highest probability as opposed toa selected threshold or only a selected threshold.

The kernel in a three dimensional image may include a height, width, anddepth. The kernel or filter may then be passed over the image todetermine if a neuron is activated and generate an activation map basedupon the presence or lack thereof of an activation, as described above.The filter or kernel K may then be determined or formulated to activatebased upon a selected feature, such an edge or other identified portion.In the current system, a gold standard or learning image set may be usedto teach the CNN the filter or kernel that identifies the selected orgold standard portion, such as the delineation, as discussed above, of avertebrae. In moving the filter over the image data it is known toconvolve or evaluate the image data based upon the filter or kernel K.

In addition to convolution layers and down sampling layers, adeconvolution layer(s) may be applied. The deconvolution layers areapplied to upsample a final segmentation map at the resolution of theoriginal image. In various embodiments, the deconvolution layers maydensify a sparse activation. The deconvolution layer may also bereferred to as a transposed convolution as described in A guide toconvolution arithmetic for deep learning, V Dumoulin, F Visin—arXivpreprint arXiv:1603.07285, arxiv.org (2016), incorporated herein byreference.

In light of the above, and with reference to FIG. 3, a CNN networkarchitecture is schematically illustrated in FIG. 3. The CNN schematicarchitecture 150 is exemplary illustrated to include an encoder part orportion that is to initially convolute and analyze the image accordingto the selected learned filters. The convoluted data may then beexpanded or decoded using a subsequent deconvolution to produce the fullresolution output 154. As discussed above, the image input 104 may bethe input for generating the CNN and trained model (i.e. when inputtingthe gold standard or user identified boundaries), and/or relating to theimage data that is attempted to segment it for identification of variousportions in the image data, as discussed further herein. In FIG. 4, theschematic architecture illustrates a deep learning CNN including aplurality of layers.

With continuing reference to FIG. 4, the input data 104 may initially beinvolved or be analyzed by a convolution or one or more convolutions(e.g. two convolutions) of a filter of 3×3×3. After the convolution abatched normalization (as discussed above) may be applied, and arectified linear unit. This may be initially applied in process 152 andgenerate a first analysis block 160 with 32 filters. A process 161 mayinclude the same steps as process 152 and generate a second analysisblock 164 with 64 filters. A max pooling step or process 170 isperformed by applying a 2×2×2 max pooling with a stride of 2 to a firstresidual block 174. The first residual block 174 may then have the twoconvolutions, a batch normalization, and a rectified linear unit appliedin a process 176 to generate a third analysis block 180. The thirdanalysis block 180 may then again have the two 3×3×3 convolutions,followed by a batch normalization, and a rectified linear unit inprocess 182 to output a fourth analysis block 186. An addition process188 of the first residual block 174 is added to the fourth analysisblock 186. This may generate a second residual block 192.

A max pooling process or operation 190 is performed on the secondresidual block 192 and then in an additional layer it is convolved bytwo 3×3×3 convolutions in process step 196 to generate the fifth block200. Again, a two 3×3×3 convolution, a batch normalization, and arectified linear unit process 204 is performed on the fifth block 200 togenerate a sixth block 204 followed again by two 3×3×3 convolutions, abatch normalization, and a rectified linear unit in process 208 to forma seventh block 212. To the seventh block 212 may be added the fifthblock 200 in an addition process 216. To the seventh block 212 isapplied a max pooling process 214 to output a third maxed pooled block220.

The third max pooled block 220 may convolved with two 3×3×3 convolutionsin process 222 to form an eighth block 224. The eighth block 224 maythen have the two 3×3×3 convolution, batched normalization, andrectified linear unit process 226 to form a ninth block 228. The ninthblock 228 may also have the two 3×3×3 convolution, batchednormalization, and rectified linear unit applied in process 232 to formthe tenth block 240. The eighth block 224 is added to the tenth block240 as a residual in the addition process 242.

In a synthesis process, the tenth block 240 is deconvoluted in adeconvolution process 246 and the seventh block 212 is concatenated inprocess 248 to form a first synthesis block 250. The first synthesisblock 250 may also then have two 3×3×3 convolutions, a batchnormalization and a rectified linear layer process 254 applied theretoto generate the second synthesis block 258. Again a two 3×3×3convolution, a batch normalization, and a rectified linear unit process260 may be done to generate the third synthesis block 262 to which adeconvolution process 266 is applied to generate a fourth synthesisblock 268 to which is added the fourth analysis block 186 in aconcatenate process 272.

The combination of the fourth analysis block 186 and the fourthsynthesis block 268 then has two convolutions of 3×3×3, a batchnormalization, and rectified linear unit process 274 to generate thefifth synthesis block 276. Fifth synthesis block 276 has the two 3×3×3convolutions, a batch normalization, and a rectified linear unitactivation process 278 to generate the sixth synthesis block 282. Thesixth synthesis block 282 is deconvoluted in a deconvolution process 284to form a seventh synthesis block 288 to which is concatenated thesecond analysis block 162 with a concatenated process 292. Thecombination block is then further applied a two 3×3×3 convolution, batchnormalization, and rectified linear unit process 296 to form the eighthsynthesis block 300 and a ninth synthesis block 304 is generated afterthe process 306 of two 3×3×3 convolutions, batch normalization andrectified linear unit process.

Finally, a 1×1×1 convolution process 310 is applied to the ninthsynthesis block. The convolution process 310 may include the soft maxactivation as discussed above. The process 310 generates the output 154that may be the output segmented image data 110, as illustrated in FIG.2. The output may include a selected number of channels, such as twochannels that may be achieved due to the soft max activation in process310. Further, the output 154 may include a 3D model or image that may bedisplayed for viewing by the user 12. Further, as illustrated in FIG. 4,the total number of filters applied in each operation is illustrated andmay include a total for the CNN 150 may be 2,400. It is understood,however, that appropriate selected number of filters may be differentthan 2,400 and may include about 100 to about 20,000,000. Accordingly,2,400 filters is disclosed herein as providing an efficient and fasterCNN 150, such as for segmentation as discussed further herein.

The schematic architecture illustrated in FIG. 4 may then be applied fortraining and/or testing or segmenting selected images. In variousembodiments, with reference to FIG. 5, a training phase process 350 isillustrated. The training phase process 350 may start with an input andmay include an image data 352 that may be appropriate image data, suchas the image data discussed above in block 104. Inputs may furtherinclude a segmentation mask, such as a binary segmentation mask 356. Thesegmentation mask 356 may be a standard or training data segmentation,such as a gold standard or user determined segmentation. For example,the binary segmentation mask may include a user (e.g. trained expert,such as a surgeon) segmentation of a selected structure, such as avertebra including the vertebrae 20 v.

After the input, including the image data 352 and the mask 356 selectedsteps may occur that included selected preprocessing. For example, anoptional resizing step in block 360 may occur to resize the image datato an appropriate or selected size. In various embodiments, voxels maybe resampled to a specific resolution, such as about 1.5 mm×1.5 mm×1.5mm. Further preprocessing may include zero padding in block 364. Asdiscussed above, zero padding may allow for insuring that an image sizeis achieved after or during the CNN process and also for insuring thatselected augmentation maintains all image data within bounds of theimage.

Selected augmentation may also be selectively applied to the image datain block 368. The augmentation in block 368 may be selectedaugmentation, such as augmentation of the input data, which may be bothor only one of offline or online. Selected off line augmentation mayinclude randomly scaling the images along a selected axis by a selectedscale factor. Scale factors may include between about 0.9 and about 1.1but may also include other appropriate scale factors. Further, imagesmay be randomly rotated around selected axes at a selected amount. Againselected amounts of rotation may include such as about minus 10 degreesto about plus 10 degrees. Online augmentation may include randomlyflipping images along different axes or transposing image channels. Theaugmentation in block 368 may assist in training the CNN by providinggreater variability in the input image data 352 than provided by theimage data set itself. As discussed above, and generally known in theart, the attempt is to have the CNN generate the filters that allow forautomatic detection of selected features, such as segmenting boundariesof vertebrae, within the image data without additional input from auser. Therefore, the CNN may better leam or more effectively leamappropriate filters by including data that is more randomized or morehighly randomized than provided by the initial image data.

The image data may then be normalized in block 272. In normalizing theimage data, the variables are standardized to have a zero mean and aunit variance. This is performed by subtracting the mean and thendividing the variables by their standard deviations.

A cropping or patch wise process may occur in block 380. In variousembodiments to achieve selected results, such as a decrease trainingtime, reduce memory requirements, and/or finer grain detail learning, aselected cropping may occur. For example, an image data of a selectedsize may be cropped, such as in half, to reduce the amount of imagetrained at a time. A corresponding portion of the segmentation mask inblock 356 is also cropped and provided in the image and mask in block380. The cropped portions may then be combined to achieve the finaloutput. The cropping process in block 380 may also reduce memoryrequirements for analyzing and/or training with a selected image dataset.

The image data, whether cropped or not from process 380 may then beinputs to the CNN in block 384. The CNN in block 384, as discussedabove, for example in FIG. 4, may then work to determine filters toachieve the output 154. The output 154 may include a probability map 388and a trained model 390 as illustrated in FIG. 5. The probability map388 is a probability of each voxel, or other selected image element,belonging to a selected labeled or delineated portion, e.g. a vertebra,a portion of a vertebra, a screw, or other selected portion in the inputimage. It is understood by one skilled in the art that the input imagemay include various selectable portions, such as a vertebra, a pluralityof vertebrae, a screw, etc. A threshold probability may be selected, invarious embodiments, for identifying or determining that a selectedimage portion is a selected portion or label. It is understood, however,that a threshold is not required and that probability map may outputselected probabilities in the output for the segmentation.

The trained model 390 includes the defined filters that may be appliedas the kernels K discussed above and may be based on the probability map388. The defined filters, as also discussed above, are used in thevarious layers to identify the important or significant portions of theimage to allow for various purposes, such as segmentation of the image.Accordingly, the trained model 390 may be trained based upon the inputimage 352 and the binary segmentation mask 356. The trained model maythen be stored or saved, such as in a memory system including thenavigation memory 72, for further access or implementation such as onthe image memory 112 and/or the navigation memory 74. The trainingprocess 350 may include various inputs, such as an amount of padding ora selected voxel size, but is generally performed by a processor systemexecuting selected instructions, such as the navigation processor system70. For example, the training of the CNN in block 384 and the trainingmodel 384 may be substantially executed by the processor system.

It is understood, however, that the trained model may also be providedon a separate memory and/or processing system to be accessed and used ata selected time. For example, the trained model may be used during aplanning phase of a procedure, and/or, during a procedure when thesubject 20 is in an operating theater during an implantation procedure.

With additional reference to FIG. 5 and turning reference to FIG. 6, thetrained model 390 from the training phase 350 may be used as an inputwhen attempting to determine a segmentation of an image, such as animage of the subject 20, in a segmentation phase 400. Accordingly,according to various embodiments, the image data of the subject 402 maybe input with the trained model 390. As discussed above inputting theimage data 402 and the trained model 390 may include accessing both theimage data and the trained model that are stored in a selected memory,such as those discussed above, by one or more of the processor systemsincluding the navigation processor system 70.

The image data 402 may be preprocessed in a manner similar to the imagedata preprocessed during the training phase 350. For example, the imagedata 402 is preprocessed in the same manner as the trained model 390 istrained. As discussed above, various preprocessing steps are optionaland may be performed on the image data 350 during the training phase.During the segmentation phase 400 the image data 402 may be or isselectively preprocessed in a similar manner. Accordingly, the imagedata 402 may be resized in block 360′, zero padding may be added inblock 364′, the image data may be normalized in block 372′. It isunderstood that the various preprocessing steps may be selected and maybe chosen during the segmentation phase 400 if performed during thetraining phase 350. The segmentation image data is the same type as thetraining image data.

After appropriate preprocessing is performed in blocks 360′, 364′, and372′, the image data 402 may be split or cropped, in block 410. Thesplitting of the image in block 410 is also optional and may be selectedbased upon processing time, memory availability, or other appropriatefeatures. Nevertheless, the image 402 may be split in a selected manner,such as along selected axes. The image data may then be merged, such asin a post processing step 414 once the segmentation has occurred.

Once the image data is preprocessed, as selected, the CNN 384, with thelearned weights and/or filters may be used to segment the image data402. The segmentation of the image data 402 by the CNN 384 may create anoutput 420 including a probability map 416 and a selected mask, such asa binary segmentation mask in the output 422. The output 420 may be anidentification of a selected geometry of the segmented portions, such asthe vertebrae 20 v or other selected features. The CNN 384, having beentaught or learned the selected features and weights, may segment theportions of the image data, such as the vertebrae.

In the output 420, probability map 416 is a probability of each voxel,or other image element or portion belonging to a selected label orportion, such as a spine, vertebra, vertebrae, screw, etc. The binarysegmentation 422 is produced from the probability map 416 by selectingall the voxels or other image portions with a probability greater than athreshold. The threshold may be any selected amount, such as about 30%to about 99%, including about 35%. It is further understood, however,that a threshold may not be required for performing the binarysegmentation 422 based on the probability map 416.

The segmentation process 400 may include various inputs, such as anamount of padding or a selected voxel size, but is generally performedby a processor system executing selected instructions, such as thenavigation processor system 70. For example, the segmentation with theCNN in block 384 and the output segmentation in block 420 may besubstantially executed by the processor system. Thus, the segmentationprocess 400, or substantial portions thereof, may be performedsubstantially automatically with the processor system executing selectedinstructions.

The output 420 may then be stored in a selected memory, such as theimaging memory 112 and/or the navigation memory 74. In variousembodiments, the segmentation output 420 may be saved in the navigationmemory 74 for use in various workflows, as discussed further herein.Moreover, the output may be output as a graphical representation, suchas one or more icons representing the geometry of the segmentedvertebrae. As illustrated in FIG. 1, the segmented portions may bedisplayed either alone or superimposed on the image 30 such as vertebraeicons or masks 20 vi′ and 20 vii′. It is understood that any appropriatenumber of segmentations may occur, and the illustration of two vertebraein FIG. 1 is merely exemplary. For example, the image data 402 may be ofan entire spine or all vertebrae of the subject 20. Accordingly, thesegmentation mask may include an identification of each of thevertebrae. Moreover, it is understood that the segmentation may be athree dimensional segmentation such that an entire three dimensionalgeometry and configuration of the vertebrae may be determined in theoutput 420 and used for various purposes, such as illustration on thedisplay 32.

The navigation system 10, as illustrated in FIG. 1, may be used forvarious purposes such as performing a procedure on the subject 20. Invarious embodiments, the procedure may include positioning an implant inthe subject, such as fixing a pedicle screw within the subject 20, suchas into one or more of the vertebrae 20 v. In performing the procedure,the tool 16 may be an implant, such as a screw. It is understood thatvarious preliminary steps may be required for performing or placing theimplant, such as passing a cannula through soft tissue of the subject20, drilling a hole into the vertebrae 20 v, tapping a hole in thevertebrae 20 v, or other appropriate procedures. It is furtherunderstood that any of the items used to perform various portions of theprocedure may be the tool 16 and that the tool 16 may also be theimplant. Any one of the portions (e.g. implants or tools or instruments)may be tracked with the respected tracking system, such assimultaneously or in sequence, and navigated with the navigation system10. During navigation, the navigation system 10 may display a positionof the tool 16 as the icon 16 i on the display 32. In a similar manner,other instruments may be navigated simultaneously with the instrument 16such that the instrument 16 may include a plurality of instruments andall or one may be individually or multiply displayed on the displaydevice 32, according to instructions such as those from the user 12.

The navigation system 10, therefore, may be used to perform theprocedure on the subject 20 by the user 12. Further, the navigationsystem 10 including the navigation processing unit 70 may executeinstructions that are stored on selected memories, such as thenavigation memory 74, for performing or assisting the user 12 inperforming a procedure on the subject 20. According to variousembodiments, such as those illustrated in FIG. 7A and FIG. 7B. Asillustrated in FIGS. 7A and 7B a workflow diagram 450 includes variousfeatures and steps that may be performed by a user, or at the directionof the user 12, and/or substantially automatically by executinginstructions by the processor system, including the navigation processorsystem 70, based upon the instruction of the user 12 or in light of aprevious step.

The workflow 450 may include a data acquisition or accessing step orportion including operating the imaging system 24, such as an O-Arm®imaging system, in block 456 to acquire an image scan or image data ofthe subject in block 460. The image data may then be accessed orreceived via a connection or communication 462. As discussed above, inrelation to FIG. 2, the image data may be accessed in block 104.Accessing or inputting data, such as the image data, in block 104 may besimilar or equivalent to the processor accessing the data 462.Similarly, as discussed further herein, the image data may be analyzedand segmented such as with the automatic segmentation process 400illustrated in FIG. 6. Regardless, the image data 460 may be acquired oraccessed by the navigation processor system 70 to allow for a workflow,including the workflow 450 to occur. The workflow 450 may furtherinclude an automated or semi-automated workflow portion 470 thatincludes analyzing the image data 460 and/or performing the portions ofthe procedure.

As discussed above the process 462 allows the image data from block 460to be used by the navigation processor system 70 according to theselected navigation process system 470. The surgical navigation system470 may include portions that are either entirely automatic (e.g.performed by a processor executing instructions according to apredetermined algorithm, a processor system executing a deep learning orneural network system, etc.) or portions that are a combination ofmanual input and automatic determination. As discussed further herein,various elements or portions may be substantially automatic unlessidentified as including a manual portion. It is understood, however,that manual portions may be not included as discussed further herein.

In the surgical navigation process 470, the navigation system 10 may beinitiated or started in block 474. The starting of the navigation systemin block 474 may be substantially manual, such as by the user 12,turning on or starting the navigation system 10. It is understood,however, that the navigation system 10 may also initiate substantiallyautomatically such as when entering a selected location, such as anoperating theater. After initiating or starting of the navigation systemin block 474, a selection of a surgeon in block 476 may be made and aselection or a procedure in block 478 may be made. It is understood thatselecting the surgeon in block 476 may be optional. Nevertheless, thenavigation memory 74 may have selected saved surgeon preferences and/oroperation that may be specific to an individual surgeon, such as theuser 12. Therefore, selecting the surgeon in block 476 may cause thenavigation processing unit 70 to access the memory 74 to select ordetermine preferences for the user 12 based upon the selection of thesurgeon in block 476. The specific surgeon may have preferences that mayaugment one or more of the following items such as specific instrumentsto be prepared for a selected procedure, a size of an implant for aselected anatomical structure, or the like. In various embodiments forexample, a selected surgeon may select to include an implant, such as apedicle screw, that has a 3 mm clearance relative to a boundary of thevertebrae while another surgeon may select to include a pedicle screwthat has a 5 mm clearance. Accordingly, the selected surgeon having theidentified preferences may be used by the processor system in executingthe processes 470 to select and/or identify selected instruments duringnavigation.

A selection of the surgeon in block 476 and the selection of theprocedure in block 478 may be substantially manually input, such asusing the input 72 or any appropriate manual input 72 to the navigationprocessing unit 70. It is understood that the user 12 may manually inputboth the selection of the surgeon in block 476 and the selection of theprocedure in block 478 or may direct that the selections in blocks 476and 478 be made. Nevertheless, it is understood that the selection ofthe surgeon in block 476 and the selection of the procedure in block 478may be substantially manual.

The navigation process 470 may automatically suggest an instrument setbased upon either one or both of the selected procedure in block 478 orthe selected surgeon in block 476. Automatically suggesting aninstrument set in block 482 may include selecting or suggestinginstrument tools, implants, or the like. For example, with regard to theplacement of a pedicle screw the navigation process 470 may suggest aninstrument (e.g. a probe, an awl, a driver, a drill tip, and a tap)and/or implant type and or geometry and size (e.g. a screw size andlength). The suggestion of an instrument and/or implant set in block 482may be based upon a selected algorithm that accesses a database ofpossible procedures and identifies tools therefrom. Further, a machinelearning system may be used to identify an instrument set based uponvarious inputs such as the procedure and surgeon, as selected surgeonsmay select different instruments and/or a surgeon's preference (e.g.pedicle screw size) may vary or change a selected instrument set.Instrument selection may also be made or assisted with heuristics basedon the segmentation as one of the inputs. Whether the instrument set isautomatically suggested in block 482 or not, instruments may be verifiedin block 484. The verification of the instruments may be insuring thatthe instruments are present in an operating theater and/or inputtingthem to the navigation system 10. For example, the navigation system 10may be instructed or used to identify a selected set of instruments ortype of instrument.

The instruments in a navigated procedure are generally tracked using aselected tracking system. It is understood that appropriate trackingsystems may be used, such as an optical or an EM tracking system asdiscussed above. In various embodiments, therefore, an instrumenttracker may be identified in block 486. An identification of theinstrument tracker may be substantially automatic based upon the trackerbeing identified by the selected tracking system, such as with theoptical localizer 82. For example, the optical localizer may be used toidentify or “view” the tracking device, such as the instrument trackingdevice 56. It is understood that a plurality of instruments may have aplurality of unique trackers on each of the instruments and therefore aviewing of a selected tracker may be used to identify the tracker to theinstrument. It is understood, however that trackers may be changeableand therefore an automatic detection may not be possible and therefore amanual identification of the instrument tracker may be selected.

A tip associated with a selected instrument may be automaticallyidentified in block 490. As discussed above, the automaticidentification of the tip may be used by “viewing” the tip with theoptical localizer 82. Accordingly, the navigation processor 70 may use adeep learning system, such as a CNN, to identify the tip relative to theinstrument and/or the tracker identified at block 486. By identifyingthe tip, the process of the procedure and the user may be assisted inidentifying selected features. Features may include a geometry of thetip used during navigation and displaying on the display device 32, suchas with the instrument icon 16 i. It is understood, however, that thetip may also be manually inputted or identified in selected procedures.

In a navigated procedure the patient 20 may also be tracked with the DRF60. In block 496 the DRF 60 may be placed or identified on the patient20. It is understood that placing the DRF 60 on a patient is generally asubstantially manual procedure being performed by the user 12 or at theinstruction of the user 12. Nevertheless, the placement of the DRF 60may also include identifying or tracking of the DRF 60 in the navigationprocess 470. Accordingly, the navigation process 470 may includetracking the DRF once placed on the patient 20. The DRF allows forregistration in block 498 to the image data input via the process 464from the image data 460. Registration can be according to anyappropriate registration process including those generally known in theart, as discussed above.

Registration allows for a subject or physical space defined by thesubject 20 to be registered to the image data such that all points inthe image data are related to a physical location. Therefore, a trackedlocation of the instrument may be displayed on the display device 32relative to the image 30. Further the registration may allow for imageportions to be registered to the patient, such as segmented portions.

Segmented portions may be segmented in block 502 (e.g. vertebrae) andsegmentations may be displayed on the display device in block 504. Asdiscussed above, segmentation of selected image portions, such asvertebrae, may be performed substantially automatically according toselected systems, such as the CNN 150, as discussed above. In additionto or alternative to the segmentation as discussed above, thesegmentation of image portions may, however, also be manual such as bythe user 12 physically tracing with a selected instrument, such as thetracked probe on the display device, on the image 30. Nevertheless, theauto-segmentation in the navigation process 470 (e.g. with thesegmentation process 400) may allow for the user 12 to not use surgicaltime or planning time to segment the vertebrae and allow for a fasterand more efficient procedure. A faster and more efficient procedure isachieved by saving the surgeon time in manual interaction with thenavigation system 10 including various software features thereof, e.g.by automatic selection of the correct tool projection based on thesegmentation.

The segmentations may also be displayed, such as display segmentationsin block 504, including the display of segmented icons 20 vi′ and 20vii′. The segmentation icons may be viewed by the user 12 and verifiedthat they overlay selected vertebrae. In addition to or as a part of averification, the image portions may also be identified and/or labeledin block 508. The labeling of the image portions may be manual, such asthe user 12 selecting and labeling each vertebra in the image 30,including the segmented portions therein. The labeling and/oridentification of the vertebrae may also be semi-automatic such as theuser 12 identifying one or less than all of the vertebrae in the image30 and the navigation processor 70 labeling all of the other vertebraerelative thereto. Finally, the identification of the vertebrae andlabeling thereof in block 508 may be substantially automatic wherein thenavigation processor 70 executes instructions, such as based upon theCNN 150, to identify selected and/or all of the vertebrae in the image30 and display labels therefore relative to the segmented portions, suchas the segmentation icons 20 vi′ and 20 vii′.

The navigation process system 470, either during a procedure or during aplanning phase, may also automatically select an implant parameter, suchas a size (e.g. length and width), in block 512. As discussed above, thevertebrae may be segmented according to selected procedures in block502. Upon segmenting the vertebrae, the dimensions of the vertebrae maybe known, including a three dimensional geometry, including size andshape. This may assist in selecting a size of an implant based upon asegmented size or determined size of the vertebrae. Also, based upon theselected surgeon preferences, based upon a selected surgeon in block476, a size of a vertebra relative to an implant may also be known andtherefore will also assist in automatically selecting the implantparameters to a specific size. The size may be output, such as on thedisplay 32 for selection and/or confirmation by the user 12.Accordingly, selecting the implant parameters, including size or othergeometry, may be made by the process system 470.

The procedure 470 may then proceed to assist in preparing and/or placinga selected implant. To place an implant, such as a pedicle screw, anentry point into the patient 20 may be determined relative to thevertebrae 20 v. With continuing reference to FIGS. 7A and 7B andadditional reference to FIG. 8 the instrument 16 may include a probe 16p with the tracker 56 (e.g. an optical tracker). The probe 16 p may bemoved relative to the subject 20, such as without piercing the subject20.

In attempting to determine an entry point in block 516 the probe 16 pmay be moved relative to the vertebrae 20 v. The vertebrae 20 v, havingbeen identified and/or labeled in block 508, may be identified basedupon a projection from the probe 56, such as from a tracked distal end520 of the probe. The probe end 520 need not puncture a soft tissue,such as a skin, of the subject 20 but rather the projection may bedetermined and/or displayed, such as with the instrument icon 16 i onthe display device 32, as illustrated in FIG. 8. The instrument icon 16i may change based upon the selected instrument and may be displayed asa projection of the probe or just a trajectory based upon the positionof the probe 16 p relative to the vertebrae 20 v. Based upon theprojection of the probe the vertebrae may be identified, such as thevertebrae 20 vi in the image 30 on the display device 32. The displaydevice 32 may display the image 30 in various manners, such as in amedial and axial view.

The projection of the instrument 16 i may be based upon a boundary ofthe vertebrae 20 v, such as based upon the segmentation of the vertebraein block 502. The segmentation may be manual, semi-manual, orsubstantially automatic (e.g. with the segmentation process 400).Nevertheless the projection icon 16 i may be limited to a boundary ofthe vertebrae 20 v and may be displayed either alone or in combinationwith the vertebrae icon 20 vi′. It is further understood that theprojection 16 i may be based upon a geometry of selected tools such as adrill so that the user 12 may view the physical extent of the drillrelative to the image 30 and the segmented vertebrae 20 vi′ to ensurethat the drill would drill far enough into the vertebrae.

The navigation process 470, as discussed above, may include portionsthat occur substantially with the navigation processor 70 either aloneor in combination with actions taken by the user 12. In variousembodiments, the find entry point features may be used to then identifyor mark a point on the skin of the subject 20. It is understood thatmarking the incision point is not required. However, performing anincision to allow other instruments to enter the subject 20 may occurafter finding the entry point as discussed above. Once an incision ismade, a tool may be navigated, such as by tracking the tool andillustrating the position of the tool on the display device 32 in block526. For example, after forming the initial incision, navigating an awlto the vertebrae identified in block 516 may occur. The tool may also bereferenced to as an instrument.

In navigating the awl relative to the vertebrae, the awl may be passedthrough the incision in the skin and contact the vertebrae. An iconrepresenting the awl or a projection from the tracked location of theawl may be illustrated relative to a vertebra at a selected time, suchas when the awl is within a selected distance to the vertebrae (e.g.less than about 1 mm to about 6 mm, including about 5 mm). Thus, theicon representing the tool may auto display a selected implant size,such as an icon superimposed on the image 30 on the display device 32 inblock 528.

Automatically display an implant size, or tool size or position, mayinclude determining the size of an implant based upon the boundaries ofthe segmented vertebrae in block 502. The navigation process 470 mayexecute instructions that based upon the segmented image geometry (e.g.including size and shape) an implant may be automatically selected anddisplayed on the display device. When the awl is at a selected positionrelative to the vertebrae displaying the automatically selected implantmay allow the user 12 to view a selected opportunity or selectedimplant. By automatically selecting the implant the user 12 need notseparately measure the vertebrae and/or trial various implants relativeto the vertebrae. Nevertheless, the user 12 may confirm and/or changethe implant size in block 532. If selected, a different size implant maythen be displayed relative to the vertebrae image 20 vi and/or thesegmentation 20 vi′. The user 12 may then view the automaticallydisplayed implant size in block 528 and/or a changed or confirmed sizein block 532.

Further the user 12 may move the tracked awl relative to the vertebrae20 v to select a position of the implant relative to the vertebrae 20 v.For example, a different position of the awl relative to the vertebrae20 v may cause the system 470 to determine or calculate a different sizeimplant (i.e. the anterior boundary of the vertebrae is further away).Once the user 12 has selected an appropriate or selected trajectory, thetrajectory may be saved. The trajectory may be saved by the user 12,such as using the input device 72. It is understood that the inputdevice, however, may be appropriate devices such as a verbal command foraudio input, a gesture, a footswitch, or the like. In addition theuser's selection may be saved based upon the selected surgeon in block476 for further or future reference.

The projection may be saved for future use and displayed and/or hiddenas selected to allow for guiding of the tapping of the vertebrae 20 v.With continuing reference to FIG. 7 and additional reference to FIGS.9A, 9B, and 9C, the vertebrae 20 v may be tapped with a tap whileviewing the display device 32 when the tap is navigated. The tap may benavigated as it is moved relative to the vertebrae 20 v by the user 12.The tap may be displayed as the icon 16 i on the display device 32relative to the image 30. Further, at a selected time, such as when thetap is near or in contact with the vertebrae 20 v a projection of atapped geometry 542 may be displayed relative to the image 30 includingthe vertebrae 20 vi as navigating the tap in block 540 to the vertebrae20 v and displaying at the projection 542 of the tapped area or volumemay allow the user 12 to confirm that the selected tapped volume basedupon a projection of the tap into the vertebrae matches the implantprojection or saved implant projection from block 536.

Once it is confirmed that the tap projection 542 matches the savedimplant projection from 536 the tap may be driven into the vertebrae 20v, as illustrated in FIG. 9B. A reduced or shrunken tap projection 544in block 546 may allow the user 12 to view the extent of the tappingrelative to the projected or selected tap length volume. The shrunkentap geometry 544 in block 546 may allow the user 12 (FIG. 9B) tounderstand the extent of the tapping performed so far and the autotapping remaining. Accordingly, the user may slow down driving of thetap into the vertebrae at a selected period while allowing for a fastand efficient tapping at an initial period.

It is understood that the processing system 70 may shrink the tappedprojection, such as the shrunken tap projection 544 in block 546,substantially automatically based upon navigation of the tap relative tothe vertebrae 20 v. The tap projection is initially based upon theselected implant projection from block 536 based upon the implant, suchas the automatically selected implant in block 528. Therefore, thenavigation process 470 may allow for efficient tapping of the vertebrae20 v by allowing the user 12 to view the tapping in process and confirmwhen the tapping is completed.

When tapping is completed, a reverse projection 550 may be automaticallydetermined and displayed in block 552. The reverse projection 550 may besubstantially equivalent or equal to the tapped depth into the vertebrae20 v and based upon the amount of tapping or depth of tapping by theuser 12. Further, the reverse projection of 550 may be substantiallyequivalent to the initial tapped projection 542, as illustrated in FIG.9A. The reverse tap projection 550 may be maintained for viewing by theuser 12 on the display device 32 relative to the vertebrae 20 vi forpositioning of the implant into the vertebrae 20 v. Moreover theinstrument icon 16 i may be a combination of both the instrument portionand a now fixed or permanent tapped projection 542′. The fixedprojection of 542′ may be initially equivalent to the reverse projection550 and allow the user 12 to view both the tapped volume (e.g. widthand/or depth) relative to the instrument 16 i and the vertebrae image 20vi.

The reverse projection may be saved in block 560 for various purposes,as discussed above for guiding or navigating the implant. Further thesaved reverse projection may be equivalent to the tapped position andmay also be saved under the selected surgeon 76 for further referenceand/or future reference.

Once the tapping of the vertebrae is performed, the implant may beplaced in the vertebrae 20 v. With continuing reference to FIGS. 7A and7B and additional reference to FIG. 10, the implant may include a screw,such as a pedicle screw, that is positioned within the vertebrae. Thescrew and a driver may be illustrated as the icon 16 i on the displaydevice 32 relative to the image 30 and the vertebrae 20 vi. The reverseprojection 550 may also be displayed in block 562 to assist innavigating the implant in block 566. As illustrated in FIG. 10, theimplant may be illustrated as at least a part of the icon 16 i such thatthe icon 16 i may be aligned with the reverse projection 550 to allowfor driving or placing the screw into the vertebrae 20 v along thetapped trajectory and volume as illustrated by the reverse projection550. Accordingly, navigating the implant in block 566 may allow the user12 to position the implant in the selected and tapped location in thevertebrae 20 v.

Tracking of the screw into the vertebrae 20 v may also allow for a savedtracked position of the screw in block 570 for the selected surgeon fromblock 476 for future use. Accordingly various features, such aspositioning of the tapped location and final position of the screw alongwith various other features, such as geometry and size of the screw maybe saved for reference of a selected surgeon for future use.

After positioning the screw by navigating the implant in block 566and/or saving the tracked screw position in block 570 a determination ofwhether further screws need be placed may be made in block 574. If noadditional implants are to be placed a NO path 576 may be followed tocomplete the procedure in block 578. Completing the procedure mayinclude decompressing vertebra, removing instrumentation from thesubject 20, closing the incision, or other appropriate features.

If it is determined that a further screw is to be implanted in block 574a YES path 582 may be followed. It is understood that the determinationof whether an additional screw is to be placed may be based upon theselected procedure from block 478 or based upon a user input from theuser 12. Accordingly, determining whether a further implant is to bepositioned in block 574 may be substantially automatic or manual.

Regardless of the procedure for determining whether a further implant isneeded, if the YES path 582 is followed an auto-switching to a furtherimage portion in block 590 may optionally occur. For example, if a firstscrew is placed in an L5 vertebra a second screw may be placed in asecond side of the L5 vertebra. Automatically switching to a separateimage or view portion of the vertebrae may assist the user 12. Further,if a second implant is positioned in the L5 vertebra and the selectedprocedure is to fuse an L5 and L4 vertebra, the image 30 mayautomatically switch to display or more closely display the L4 vertebrafor further procedure steps. Accordingly, auto-switching to anotherimage portion in block 590 may assist the user 12 in efficientlyperforming the procedure.

Whether an optional automatic switching to additional image step isperformed or not, the determination that further implants are to beplaced may allow the navigation process 470 to loop to block 512 toautomatically select parameters of the next implant and continue theprocedure from there. It is understood that various other portions mayalso be repeated, such as identifying instruments or tips (e.g. blocks486 and 490), but such application may not be required, particularly ifthe instrumentation maintains or remains the same from a first implantto additional implants. Nevertheless, a selected number of implants maybe positioned in the subject 12 by continuing the process from block 512to the decision block 574 until no further implants are determined to benecessary or part of the procedure and the no path 576 may be followed.

Accordingly, in light of the above, the navigation process 470 may beused to place implants in the subject 20, such as spinal implants, in aselected and efficient manner by allowing the processor system to assistthe user 12 in performing various procedures and/or selecting variousportions of the procedure, such as implant geometry and/or positioning,automatically. The user 12 may then use the selections as an initialstarting point or confirm that the suggestions are appropriate andcontinue with the procedure. Accordingly a cold or “blank slate” is notrequired to perform the procedure by the user 12.

Example embodiments are provided so that this disclosure will bethorough, and will fully convey the scope to those who are skilled inthe art. Numerous specific details are set forth such as examples ofspecific components, devices, and methods, to provide a thoroughunderstanding of embodiments of the present disclosure. It will beapparent to those skilled in the art that specific details need not beemployed, that example embodiments may be embodied in many differentforms and that neither should be construed to limit the scope of thedisclosure. In some example embodiments, well-known processes,well-known device structures, and well-known technologies are notdescribed in detail.

The foregoing description of the embodiments has been provided forpurposes of illustration and description. It is not intended to beexhaustive or to limit the disclosure. Individual elements or featuresof a particular embodiment are generally not limited to that particularembodiment, but, where applicable, are interchangeable and can be usedin a selected embodiment, even if not specifically shown or described.The same may also be varied in many ways. Such variations are not to beregarded as a departure from the disclosure, and all such modificationsare intended to be included within the scope of the disclosure.

What is claimed is:
 1. A surgical navigation system configured foroperation in a procedure, comprising: a memory system having storedthereon: (i) a convolutional neural network (CNN) configured foranalyzing an image data, (ii) a trained model for use with the CNN thatis trained to segment at least one spinal vertebrae in the image data,(iii) wherein the image data includes an acquired image of a subject; aprocessor system configured to: access the memory system to execute theCNN and the trained model to segment the image data; output athree-dimensional (3D) image data of the segmentation; based on theoutput 3D image data automatically output a workflow procedure portionfor a surgical procedure; wherein the CNN includes at least one of alimited number of parameters or selected weights on parameters toincrease a probability of correct segmentation; wherein the processorsystem is configured to further automatically determine a tool extensionand a displayed tool icon relating to the automatically determined toolextension; wherein the tool icon displayed on a screen includes aprojection from a tracked distal tip of the tool to a portion of theimage; wherein the projection is automatically determined and displayedbased on the output 3D image data segmentation from the tracked distaltip of the tool.
 2. The system of claim 1, wherein the image data isacquired with a x-ray cone-beam imager; wherein the CNN is a multiplelayer neural network.
 3. The system of claim 2, wherein the CNN furtherincludes a residual block included in a synthesis step from an analysisstep in the same layer.
 4. The system of claim 1 wherein thesegmentation delineates at least one of the vertebrae in the image data.5. The system of claim 1, wherein the segmentation delineates all of thevertebrae in the image data.
 6. The system of claim 1, furthercomprising: a display device to display; wherein the processor isconfigured to display the output 3D image data segmentation.
 7. Thesystem of claim 6, wherein the displayed output 3D image datasegmentation includes at least one mask displayed as a graphicalrepresentation overlayed on an image generated from the image data. 8.The system of claim 1, wherein the processor is further configured toexecute instructions to automatically calculate at least onecharacteristic of appropriate tools for a surgery based on a determinedsize of the at least one spinal vertebrae in the output 3D image datasegmentation.
 9. The system of claim 8, wherein the at least onecharacteristic includes a physical dimension.
 10. The system of claim 1,wherein the processor system is configured to further automaticallydetermine at least one of a screw size and a displayed screw iconrelating the automatically determined screw size.
 11. The system ofclaim 1, wherein the output 3D image data segmentation includes adelineated area of the at least one spinal vertebrae.
 12. The system ofclaim 1, wherein the processor is further configured to automaticallycalculate and output at least one characteristic of an appropriateimplant based on the output 3D image data segmentation; wherein theoutput 3D image data segmentation includes at least one of a size of ananatomical portion present in the output 3D image data segmentation. 13.The system of claim 12, wherein the size includes at least one of alength or a width of a pedicle screw.
 14. The system of claim 1, whereinthe processor is further configured to automatically determine a size ofan icon for display on the display device based at least on the output3D image data segmentation.
 15. The system of claim 1, wherein theprocessor is further configured to automatically determine anidentification of a selected one vertebrae of a plurality of vertebraebased on the output 3D image data segmentation and a tracked position ofa probe; wherein the probe is configured to be moved by a user.
 16. Thesystem of claim 1, wherein the processor is further configured toautomatically determine a reverse projection of a tapping of at leastone vertebrae that relates to the output 3D image data segmentation of avertebrae of a spine; wherein the reverse projection is displayed as anicon for viewing by a user during implantation of an implant to guideplacement of the implant.
 17. The system of claim 1, wherein theprocessor is further configured to automatically determine at least asecond vertebrae for placement of a second implant after completingplacement of a first implant in a first vertebrae.
 18. A surgicalnavigation system configured for operation in a procedure, comprising: atracking system configured to track at least a patient tracking deviceand an instrument tracking device; a memory system having storedthereon: (i) an image data of at least a first vertebrae; (ii) athree-dimensional (3D) image data of a segmentation of the at leastfirst vertebrae; a display device to display at least one of the imagedata of at least a first vertebrae or the 3D image data of thesegmentation of the at least first vertebrae; a processor systemconfigured to: determine a position of an instrument based on trackingthe instrument tracking device; register the image data to a patientspace; display on the display device an icon representing at least aprojection of a trajectory from the instrument; and based on the output3D image data of the segmentation automatically output a workflowprocedure portion for a surgical procedure.
 19. The system of claim 18,wherein the output 3D image data of the segmentation includes an imageconfigured to be displayed with the display device.
 20. The system ofclaim 19, wherein the output 3D image data of the segmentation isdetermined automatically by a convolutional neural network having beentrained to delineate at least the first vertebrae.
 21. The system ofclaim 18, wherein the output 3D image data of the segmentationdelineates all of the vertebrae in the image data.
 22. The system ofclaim 18, wherein the processor is further configured to automaticallydetermine a length of the projection based on the determined trajectoryand a determined size of at least the first vertebrae in the output 3Dimage data of the segmentation.
 23. The system of claim 18, wherein theprocessor system is configured to further automatically determine a toolsize and a displayed tool icon relating the automatically determinedtool size; wherein the automatically determined tool size is based onthe output 3D image data of the segmentation.
 24. The system of claim23, wherein the tool icon displayed on the screen includes a projectionfrom a tracked distal tip of the instrument to a portion of the image;wherein the projection is automatically determined and displayed basedon the output 3D image data segmentation from the tracked distal tip ofthe instrument.
 25. The system of claim 18, wherein the processor systemis further configured to automatically determine and output at least onecharacteristic of an implant based on the output 3D image data of thesegmentation; wherein the output 3D image data of the segmentationincludes at least one of a size or a shape of an anatomical portionpresent in the output 3D image data of the segmentation.
 26. The systemof claim 18, wherein the processor system is further configured toautomatically determine an identification of at least the firstvertebrae based on the output 3D image data of the segmentation and atracked position of a probe; wherein the probe is configured to be movedby a user.
 27. The system of claim 18, wherein the processor system isfurther configured to automatically determine a reverse projection basedon tracking the instrument within at least the first vertebrae thatrelates to the output 3D image data of the segmentation of a vertebra ofa spine; wherein the reverse projection is displayed as an icon forviewing by a user during implantation of an implant to guide placementof the implant.
 28. The system of claim 18, wherein the processor systemis further configured to automatically determine at least a secondvertebrae for placement of a second implant after completing placementof a first implant in a first vertebrae.
 29. A method of performing asurgical procedure with a surgical navigation system, comprising:accessing a trained model from a memory system; accessing an image dataof a subject, wherein the image data includes a cone beam computedtomography, analyzing the accessed image data with a convolutionalneural network (CNN) configured for analyzing an image data with thetrained model; outputting a three-dimensional (3D) image data of asegmentation based on the analysis of the image data with the CNN;automatically outputting a workflow procedure portion for a surgicalprocedure based on the output 3D image data of the segmentation;automatically determining an implant size; and displaying theautomatically determined implant size with a display device relative toan image of the subject based on the output 3D image data of thesegmentation; wherein the automatically determined implant size is basedon the output 3D image data of the segmentation; wherein the CNNincludes at least one of a limited number of parameters or selectedweights on parameters to increase a probability of correct segmentation.30. The method of claim 29, wherein the output 3D image data of thesegmentation includes an image configured to be displayed with thedisplay device.
 31. The system of claim 30, wherein the output 3D imagedata of the segmentation is determined automatically by a convolutionalneural network having been trained to delineate at least a firstvertebra.
 32. A method of performing a surgical procedure with asurgical navigation system, comprising: accessing a trained model from amemory system; accessing an image data of a subject, wherein the imagedata includes a cone beam computed tomography; analyzing the accessedimage data with a convolutional neural network (CNN) configured foranalyzing an image data with the trained model; outputting athree-dimensional (3D) image data of a segmentation based on theanalysis of the image data with the CNN; and automatically outputting aworkflow procedure portion for a surgical procedure based on the output3D image data of the segmentation; tracking an instrument relative tothe subject to determine a trajectory from a distal end of theinstrument into the subject; and automatically determining a length of aprojection for display with a display device relative at least adelineated first vertebrae; wherein the length of the projection isbased on the determined trajectory and a determined size of at least thedelineated first vertebrae in the output 3D image data of thesegmentation; wherein the CNN includes at least one of a limited numberof parameters or selected weights on parameters to increase aprobability of correct segmentation.
 33. The system of claim 29, furthercomprising: automatically determining an identification of at leastdelineated first vertebrae in the output 3D image data of thesegmentation and a tracked position of a probe; wherein the probe isconfigured to be moved by a user.
 34. The system of claim 29, furthercomprising: automatically determining a reverse projection based ontracking an instrument within at least a first vertebrae that relates toat least a delineated first vertebrae in the output 3D image data of thesegmentation; wherein the reverse projection is displayed as an icon forviewing by a user during implantation of an implant to guide placementof the implant.
 35. The system of claim 18, wherein the wherein theprocessor system is further configured to automatically determine atleast a second vertebrae for placement of a second implant aftercompleting placement of a first implant in at least the delineated firstvertebrae in the output 3D image data of the segmentation.